Skip to content

Conversation

@drbenvincent
Copy link
Collaborator

@drbenvincent drbenvincent commented Nov 3, 2025

Adds a new notebook demonstrating how to conduct lift tests using Interrupted Time Series analysis when control groups are unavailable (e.g., national-level campaigns).

Key features:

  • Simulates realistic marketing data with adstock and saturation effects
  • Shows how to run ITS analysis to measure campaign lift
  • Extracts mean and standard deviation outputs for MMM calibration
  • Calculates absolute lift, percentage lift, and ROI with uncertainty
  • Includes guidance on formatting results for PyMC-Marketing integration

This bridges the gap between experimental lift testing and MMM calibration, providing the workflow for generating experimental evidence that can improve attribution models.


📚 Documentation preview 📚: https://causalpy--547.org.readthedocs.build/en/547/

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@drbenvincent drbenvincent marked this pull request as draft November 3, 2025 11:29
@codecov
Copy link

codecov bot commented Nov 3, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.50%. Comparing base (828ba2e) to head (dc21e9c).
⚠️ Report is 40 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #547      +/-   ##
==========================================
+ Coverage   95.59%   96.50%   +0.90%     
==========================================
  Files          29       31       +2     
  Lines        2681     4173    +1492     
==========================================
+ Hits         2563     4027    +1464     
- Misses        118      146      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@drbenvincent
Copy link
Collaborator Author

@cetagostini I think I should be adding in the channel spend variables into the model formula, right?

@review-notebook-app
Copy link

review-notebook-app bot commented Nov 5, 2025

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2025-11-05T07:17:35Z
----------------------------------------------------------------

All good, but two important points:

  1. The spend or response should not be scaled. So, you can safely pass everything in the original scale, the model will handle everything under the hood. Doc here: https://github.com/pymc-labs/pymc-marketing/blob/main/pymc_marketing/mmm/multidimensional.py#L1828
  2. The impact input should be MEAN not cumulative. Basically, this same values divide by the N length of the experiment.

drbenvincent commented on 2025-11-13T14:11:05Z
----------------------------------------------------------------

Removed mention of normalising the results in cb64c19

drbenvincent commented on 2025-11-13T14:28:55Z
----------------------------------------------------------------

remaining stuff about the mean (lift/week of the experiment) should be done in dc21e9c

@review-notebook-app
Copy link

review-notebook-app bot commented Nov 5, 2025

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2025-11-05T07:17:36Z
----------------------------------------------------------------

Good, I'll make a PR to make things clear there as well.


drbenvincent commented on 2025-11-06T09:47:21Z
----------------------------------------------------------------

I've already given that a go in a PR that I think you've seen pymc-labs/pymc-marketing#2061

@drbenvincent drbenvincent added the documentation Improvements or additions to documentation label Nov 6, 2025
Copy link
Collaborator Author

I've already given that a go in a PR that I think you've seen pymc-labs/pymc-marketing#2061


View entire conversation on ReviewNB

@drbenvincent drbenvincent added this to the 0.6.0 milestone Nov 13, 2025
Copy link
Collaborator Author

Removed mention of normalising the results in cb64c19


View entire conversation on ReviewNB

Copy link
Collaborator Author

remaining stuff about the mean (lift/week of the experiment) should be done in dc21e9c


View entire conversation on ReviewNB

@drbenvincent drbenvincent marked this pull request as ready for review November 13, 2025 15:10
@drbenvincent drbenvincent merged commit 860c1bc into main Nov 14, 2025
10 checks passed
@drbenvincent drbenvincent deleted the its-lift-test branch November 14, 2025 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants